Linear Regression

Before we start

__Please load the following packages

library(tidyverse)
library(MASS)
library(ISLR)
library(ISLR2)

Access lecture slide from bit.ly/aar-ug

Hello

Advertising Data

Association between slaes and budget?

A simple linear model

\[ Y \approx \beta_0 + \beta_1X \]

\[\beta_0\hspace{1mm} is\hspace{1mm}population\hspace{1mm}intercept\]

\[\beta_1\hspace{1mm} is\hspace{1mm}population\hspace{1mm}slope\] Our estimates are represented as :

\[ \hat\beta_0\] \[\hat\beta_1\]

How to reach the best estimate?

The Idea is to, essentially, draw a line through the points such that distance of every point from line is as small a possible.

Least squares

One way to get estimates of population coefficients or parameters is minimizing least squares.

\[sales \approx \beta_0 + \beta_1*TV\]

\[\hat y_i = \hat\beta_0 + \hat\beta_1x_i\] \[e_i = y_i - \hat y_i\]

\[RSS = e_1^2 + e_2^2....+e_n^2\]

Minimize RSS

Least square coefficient estimates

\[ \hat\beta_1 = \frac{\sum_i^n(x_i - \bar x)(y_i - \bar y)}{\sum_i^n(x_i - \bar x)^2} \]

\[ \hat\beta_0 = \bar y - \hat\beta_1\bar x \]

The model

Call:
lm(formula = sales ~ TV, data = advertisement)

Coefficients:
(Intercept)           TV  
    7.03259      0.04754  

“For every`additional $1000 spent on TV advertisement budget, there is additional sale of ~47.5 units”

Exercise

Use the data Auto from the {ISRL2} Fit this model.

\[horsepower = \beta_0 + \beta_1*weight + \epsilon\]

find coeff estimates and residuals: \[\hat\beta_0\] and \[\hat\beta_1\]

Fishy business

The F-statistic

\[ F = \frac{(TSS-RSS)/p}{RSS/(n-p-1)} \] In a multiple linear regression we need to ensure that all coefficients for \(p\) predictors are not equal to zero. Meaning that at least one of those is non-zero. F-statistic helps us ensure that.

That means if \(H_0\) is true, F will be near 1 and if not F will be far larger than 1.

Variable Selection

Well if one or more predictors have non-zero coef and are associated with response, Which are those?

• We will begin with trying out various subsets of predictors and assess all possible models.(Can you try out \(2^p\) models?)
• Mallow’s \(C_p\), Akaike information criterion (AIC), Bayesian information criterion (BIC), and adjusted \(R^2\).
• Forward selection, Backward Selection

Qualitative Variables - dummies

Exercise

Use Credit in {ISLR2} data, use balance as response and create a model that you think is good. Try out qualitative variables as well. Begin by looking at all possible variables in data and think which could affect the response.

TRY the {broom} package as well.

Extending the linear model

• Additive property (\(X_j\) may be associated with response in a manner that changes with different values of other predictors )
• Linear property (Response will experience the same amount of changes with one unit change in a predictor, regardless of the value of predictor.)

We will see some “classic” ways of relaxing these properties.

Breaking Additive Assumption by Interaction

Instead of \(Y = \beta_0 + \beta_1X_1 + \beta_2X_2\)

We add a interaction term for \(X_1\) and \(X_2\)

\[ \begin{align} \begin{aligned} Y & = \beta_0 + \beta_1X_1 + \beta_2X_2 + \beta_3X_1X_2\\ & = \beta_0 + X_1(\beta_1 + \beta_3X_2)+\beta_2X_2 \end{aligned} \end{align} \]

What to do when p-values of \(\beta_1\) and \(\beta_2\) are large and the p-value of the interaction term is small? (hierarchical principle)

Polynomial Regression

# A tibble: 1 × 12
  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>
1     0.688         0.686  4.37      428. 5.40e-99     2 -1133. 2274. 2290.
# ℹ 3 more variables: deviance <dbl>, df.residual <int>, nobs <int>

# A tibble: 1 × 12
  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>
1     0.606         0.605  4.91      600. 7.03e-81     1 -1179. 2363. 2375.
# ℹ 3 more variables: deviance <dbl>, df.residual <int>, nobs <int>

Potential Problems

Potential problems - Non linear Data

Residual vs Fitted, if there is indication of non-linear relation, try non-linear transformations on you predictors.

Potential Problems - Correlated Error Terms

• We assume that \(\epsilon_1, \epsilon_2, ...\epsilon_n\) are uncorrelated. This means that on reasonable deduction can be done about \(\epsilon_{n+1}\) from the information we have about \(\epsilon_n\).
• What if they are?
• recall that \(SE\) is calculated based on this assumption.
• This means that in case there is correlation in error terms, we may end up trusting the model more than we should.
• This is often seem in time-series
• How to test - Durbin-Watson test, Ljung-Box Q test

Potential Problems - Non-Constant Variance of error term

• This means that \(Var(\epsilon_i)\) could increase or decrease with the response.
• If this is found, this could overestimate \(SE\).
• Can be identified with a funnel shape of the residual vs fit chart.
• To address this, use of a concave function, e.g. log(Y) or \(\sqrt{Y}\).

Potential Problem - Outlier

“An outlier is a point for which \(Y_i\) is far from the value predicted by the model.”

• If the predictor value for an Outlier is usual, then it has little effect on the fit, but can effect RSE, p-values and \(R^2\).
• We can use residual vs fitted chart to identify an Outlier. But the question remains: How high is high?
• To answer that, plot studentized residuals. How are these calculated? studres(model). For each residual this value is expected to be less that absolute 3, i.e. between -3 and 3.